fix: extend zero_division parameter to percentage and range-based metrics#3122
fix: extend zero_division parameter to percentage and range-based metrics#3122mahimn01 wants to merge 1 commit into
Conversation
3565785 to
d55581a
Compare
|
Hi @dennisbader — gentle nudge on this when you get a chance. CI hasn't triggered yet (looks like it needs a maintainer approve-and-run for first-time contributors). It extends the #3059 |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3122 +/- ##
==========================================
- Coverage 96.55% 96.48% -0.07%
==========================================
Files 160 160
Lines 17285 17293 +8
==========================================
- Hits 16689 16686 -3
- Misses 596 607 +11 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Hey @mahimn01, thanks for this PR and contributing :). Will review it shortly as this is something we should definitely add, but it would be also great if you could create an issue for this first. Thanks! |
jakubchlapek
left a comment
There was a problem hiding this comment.
Hey @mahimn01, I've taken an initial look at the code, it looks fine, thanks. The ope clarification makes sense to me. One thing that we would definitely need to add is analogous support to all the percentage metrics (e.g. ape, mape, sape, smape, wmape, ope, arre, marre).
Before reviewing further I would like to discuss the fill semantics. For the scaled metrics the user could pass in the insample metric leading to the zero_division issue in the first place. Here as we don't supply that, there is no issue of matching the periodic series with forecasts. To align this to the existing solution I think what would make more sense than returning nans is returning a perfect forecast (e.g. 0/0 for MAPE returns 0% error). This would then depend on the metric and best score, so we can think on how to structure this best (maybe collapse it with the _safe_scaled_divide as the logic will be quite similar I think.) Let me know what you think on this :)
|
Thanks @jakubchlapek, this makes sense — best-score for 0/0 is the right call, and it actually fixes a latent bug I'd missed: with NaN-everywhere, a genuinely perfect forecast on a degenerate series scores NaN ( I'll collapse On scope (full breakdown in the issue): One small flag: Inline replies:
I've opened #3132 to track the design. Thanks for the thoughtful review! |
|
Hey @jakubchlapek — no rush, just a bump on this one whenever you get a chance. I think we're aligned on the core: fold the percentage path into the unified Two calls are yours before I finish the rework:
Point me either way and I'll turn it around quickly, and rebase at the same time since master's moved. |
|
Hey @mahimn01, thanks for the update. Regarding the |
|
Hey @jakubchlapek, agreed on both. On ape/mape, you're right. Once On MAPE's default, ape and mape currently raise on a zero actual (it's documented), so defaulting I'll unify |
7e13644 to
c72b6bb
Compare
|
Reworked and pushed, also rebased onto master. One heads-up, since mape is the default metric for |
c72b6bb to
332f203
Compare
Add zero_division to ape, mape, sape, smape, wmape, ope, arre, marre, and coefficient_of_variation. Share denominator handling with scaled metrics while preserving their near-zero behavior. Percentage and range metrics use exact zeros: 0/0 returns 0.0, nonzero/0 returns NaN, and raise keeps explicit failure behavior. Handle zero ranges element-wise, allow negative OPE sums, and keep missing-only WMAPE and OPE inputs as NaN.
332f203 to
0f6cddd
Compare
Checklist before merging this PR:
Closes #3132.
Summary
Adds consistent
zero_divisionhandling toape,mape,sape,smape,wmape,ope,arre,marre, andcoefficient_of_variation."warn"(default):0/0returns0.0, while non-zero/0 returnsnp.nan, and logs a warning."raise": raises aValueError.These metrics reuse
_safe_scaled_dividewith a configurable zero fill. Scaled metrics keep their existing near-zero behavior; percentage and range metrics only treat exact zeros as zero.Also:
openow accepts a negativeactual_seriessum.arreandmarrehandle zero-range components element-wise.wmapeandopeinputs remainnp.nan.Tests
uv run pytest darts/tests/metrics/test_metrics.py— 478 passeduv run pre-commit run --all-files— passeduv run make --directory ./docs build-all-docs— passed